VSR: A Unified Framework for Document Layout Analysis Combining Vision, Semantics and Relations

Document layout analysis is crucial for understanding document structures. On this task, vision and semantics of documents, relations between components contribute to the process. Though many works have been proposed exploit above information, they show unsatisfactory results. NLP-based methods model as a sequence labeling task insufficient capabilities in modeling. CV-based detection or segmentation but bear limitations inefficient modality fusion lack relation modeling components. To address limitations, we propose unified framework VSR analysis, combining vision, relations. supports both methods. Specifically, first introduce through image text embedding maps. Then, modality-specific visual semantic features are extracted using two-stream network, which adaptively fused make full use complementary information. Finally, given component candidates, module based on graph neural network incorported output final three popular benchmarks, outperforms previous models by large margins. Code will be released soon.